feat(pdf): refactor MinerU parsing to the official file_parse API - #3953
feat(pdf): refactor MinerU parsing to the official file_parse API#3953zonas0574 wants to merge 3 commits into
Conversation
|
thanks, our team will review this |
| "pdf": { | ||
| "strategy": "auto", | ||
| "mineru_endpoint": "http://127.0.0.1:8000", | ||
| "mineru_api_key": "your-api-key", |
There was a problem hiding this comment.
这里方便提供下使用的是哪个版本的开源镜像服务吗?MinerU 3.4.4 提到的鉴权是MinerU -> vLLM 的下游鉴权,这里的api-key 是vllm部分的输入?
There was a problem hiding this comment.
对是3.4.4,但这个其实目前没有用,我看之前在我也没删就保留,要不我先删了 = =
There was a problem hiding this comment.
对是3.4.4,但这个其实目前没有用,我看之前在我也没删就保留,要不我先删了 = =
之前的应该是有开发者支持了一版调用mineru官方api的方式,里面会有apikey存在。可以看看如何兼容官方api和自运营api方式哈,比如加个字段区分?
There was a problem hiding this comment.
当 PDF 解析策略为 mineru 时,MinerU 是必需依赖;当策略为 auto 且配置了
mineru_endpoint 时,它也承担本地解析失败后的回退职责。目前在实际解析 PDF 时才
调用 /file_parse,因此 endpoint 配置错误、服务未启动或协议不兼容可能会延后到运
行期才发现。
这里是否可以考虑在 OpenViking 初始化完成前增加一个 MinerU 启动预检?例如:
- 对 {mineru_endpoint}/health 做一次短时轮询。
- 收到 HTTP 200、合法 JSON 且包含 protocol_version 时视为就绪。
- strategy="mineru",或 strategy="auto" 且配置了 mineru_endpoint 时执行。
- strategy="local",以及未配置 endpoint 的 auto 模式可跳过。
- 若预检超时,启动时给出包含 endpoint、最后错误和修复建议的提示。
这样可以更早发现依赖配置问题,也避免首个 PDF 导入任务才暴露错误
示例
import asyncio
import time
import httpx
async def wait_for_mineru_ready(endpoint: str, timeout: float = 5.0) -> None:
health_url = f"{endpoint.rstrip('/')}/health"
deadline = time.monotonic() + timeout
last_error = "no response"
async with httpx.AsyncClient(timeout=0.5) as client:
while time.monotonic() < deadline:
try:
response = await client.get(health_url)
payload = response.json()
if (
response.status_code == 200
and isinstance(payload, dict)
and payload.get("protocol_version")
):
return
last_error = (
f"unexpected response: HTTP {response.status_code}, "
f"protocol_version={payload.get('protocol_version')!r}"
)
except (httpx.HTTPError, ValueError) as exc:
last_error = str(exc)
await asyncio.sleep(0.2)
raise RuntimeError(
f"MinerU startup preflight failed for {health_url} after {timeout}s. "
f"Last error: {last_error}. "
"Start a compatible mineru-api service or correct
pdf.mineru_endpoint."
)
在服务初始化流程中、设置 _initialized = True 前调用:
pdf_config = self._config.pdf
should_preflight_mineru = (
pdf_config.strategy == "mineru"
or (
pdf_config.strategy == "auto"
and pdf_config.mineru_endpoint is not None
)
)
if should_preflight_mineru and pdf_config.mineru_endpoint:
await wait_for_mineru_ready(pdf_config.mineru_endpoint)
There was a problem hiding this comment.
对是3.4.4,但这个其实目前没有用,我看之前在我也没删就保留,要不我先删了 = =
之前的应该是有开发者支持了一版调用mineru官方api的方式,里面会有apikey存在。可以看看如何兼容官方api和自运营api方式哈,比如加个字段区分?
官方我看不支持file_parse,只有异步提交任务,然后查询任务,和现在方式不是特别契合,不知道是不是可以和知识库解析一样变成队列
Co-Authored-By: Claude <noreply@anthropic.com>
|
之前专注在pdf.py中,对全局内容考虑不周,现在按建议补上,具体如下
|
Description
支持最新本地mineru部署的解析API
Human Involvement
Related Issue
#3910
Type of Change
Changes Made
Testing
Checklist
Screenshots (if applicable)
Additional Notes
不支持官方云平台https://mineru.net/apiManage/docs (只有异步接口),仅支持开源本地部署版本